plot(): analyze distributions

Overview

The function plot() explores the distributions and statistics of the dataset. It generates a variety of visualizations and statistics which enables the user to achieve a comprehensive understanding of the column distributions and their relationships. The following describes the functionality of plot() for a given dataframe df.

1. plot(df): plots the distribution of each column and computes dataset statistics

2. plot(df, x): plots the distribution of column x in various ways, and computes its statistics

3. plot(df, x, y): generates plots depicting the relationship between columns x and y

The generated plots are different for numerical and categorical columns. The following table summarizes the output for the different column types.

x	y	Output
None	None	dataset statistics, histogram or bar chart for each column
Numerical	None	column statistics, histogram, kde plot, qq-normal plot, box plot
Categorical	None	column statistics, bar chart, pie chart, word cloud, word frequencies
Numerical	Numerical	scatter plot, hexbin plot, binned box plot
Numerical	Categorical	categorical box plot, multi-line chart
Categorical	Numerical	categorical box plot, multi-line chart
Categorical	Categorical	nested bar chart, stacked bar chart, heat map

Next, we demonstrate the functionality of plot().

Load the dataset

dataprep.eda supports Pandas and Dask dataframes. Here, we will load the well-known adult dataset into a Pandas dataframe.

import pandas as pd
df = pd.read_csv("https://www.openml.org/data/get_csv/1595261/phpMawTba", na_values = [' ?'])

Get an overview of the dataset with plot(df)

We start by calling plot(df) which computes dataset-level statistics, a histogram for each numerical column, and a bar chart for each categorical column. The number of bins in the histogram can be specified with the parameter bins, and the number of categories in the bar chart can be specified with the parameter ngroups. If a column contains missing values, the percent of missing values is shown in the title and ignored when generating the plots.

from dataprep.eda import plot
plot(df)

NumExpr defaulting to 4 threads.

Stats and Insights

Dataset Statistics

Number of Variables	15
Number of Rows	48842
Missing Cells	6465
Missing Cells (%)	0.9%
Duplicate Rows	52
Duplicate Rows (%)	0.1%
Total Size in Memory	30.1 MB
Average Row Size in Memory	645.7 B
Variable Types	Numerical: 6 Categorical: 9

Dataset Insights

workclass has 2799 (5.73%) missing values	Missing
occupation has 2809 (5.75%) missing values	Missing
native-country has 857 (1.75%) missing values	Missing
age is normally distributed	Normal
education-num is normally distributed	Normal
capital-gain has 44807 (91.74%) zeros	Zeros
capital-loss has 46560 (95.33%) zeros	Zeros

age

Column Insights

1. age is normally distributed

workclass

Column Insights

1. workclass has 2799 (5.73%) missing values

fnlwgt

education

education-num

Column Insights

1. education-num is normally distributed

marital-status

occupation

Column Insights

1. occupation has 2809 (5.75%) missing values

relationship

race

sex

capital-gain

Column Insights

1. capital-gain has 44807 (91.74%) zeros

capital-loss

Column Insights

1. capital-loss has 46560 (95.33%) zeros

hours-per-week

native-country

Column Insights

1. native-country has 857 (1.75%) missing values

class

Understand a column with plot(df, x)

After getting an overview of the dataset, we can thoroughly investigate a column of interest x using plot(df, x). The output is of plot(df, x) is different for numerical and categorical columns.

When x is a numerical column, it computes column statistics, and generates a histogram, kde plot, box plot and qq-normal plot:

plot(df, "age")

When x is a categorical column, it computes column statistics, and plots a bar chart and pie chart:

plot(df, "education")

Understand the relationship between two columns with plot(df, x, y)

Next, we can explore the relationship between columns x and y using plot(df, x, y). The output depends on the types of the columns.

When x and y are both numerical columns, it generates a scatter plot, hexbin plot and box plot:

plot(df, "age", "hours-per-week")

When x and y are both categorical columns, it plots a nested bar chart, stacked bar chart and heat map:

plot(df, "education", "marital-status")

When x and y are one each of type numerical and categorical, it generates a box plot per category and a multi-line chart:

plot(df, "age", "education")
# or plot(df, "education", "age")